After completing this lesson, you’ll be able to:
In this lesson, you will:
Transformers that filter don’t transform data content, yet they are the most commonly used type of transformer!
Filtering is the technique of subdividing data as it flows through a workspace. It occurs when a transformer has multiple output connections, each carrying data that needs to be processed differently. Here (for example), the incoming stream of records A is filtered into two new streams, B and C, using a transformer with two output ports:
A filtering transformer may be any transformer with multiple output ports, such as the GeometryFilter or Sampler transformers, the latter of which creates a sample selection of data and separates it through Sampled and NotSampled output ports:
However, these are mostly in-built, fixed tests. Conditional filtering is where a user-defined test or condition decides which records are output to which connection. The Tester transformer is the most obvious example of this. It carries out a test with different output ports for records that pass and fail the test.
Transformers in FME allow you to filter records in several ways. A significant distinction is between transformers that filter based on attribute values and those that filter based on spatial data. An example of attribute filtering is filtering records into two streams based on the date value stored in an attribute called LastModified. An example of spatial filtering is filtering retail locations within 750 meters of a transit station.
This course covers attribute filtering only. Complete the Analyze Spatial Data course or read Common GIS Operations to learn about spatial filtering.
Many transformers in the Filters and Joins category can carry out tests and redirect data according to the results:
Although the Tester transformer is the most used in this category, there are many related transformers, such as the TestFilter, GeometryFilter, AttributeFilter, SpatialFilter, and Sampler.
The Tester and TestFilter are the two critical transformers for conditional filtering. They test the values on attribute values.
The Tester transformer is generally for single tests that produce a Yes/No result.
For example, here we wish to decide whether to send out snow plows to a particular road based on whether the value of the Snowfall attribute is greater than 150 mm (approximately 6 inches):
If snowfall exceeds 150 mm, the road record passes the test, and the workspace will recommend sending snow plows.
Multiple Clauses
Each clause in the Tester is an individual test that allows a Passed/Failed result. For example, each of the following criteria might be a separate test:
However, the Tester allows multiple tests, where a user can combine any number of clauses using an AND and OR statement. So, instead of individual tests, I might ask:
The Tester also allows mixing AND and OR statements using a Composite Test. For example:
However complex the test becomes, it still results in a single Yes/No (binary) result; records will either pass or fail this set of tests.
It's also worth remembering that we aren't restricted to simple tests of equality (A=B); in the above, there are also "greater than" and "less than" tests. That's because you can use many different operators in a test clause.
You can specify the value of Logic from a dropdown list or type values in directly. That means you aren't limited to single parentheses, as the dropdown field would lead you to believe!
The overall logic of the conditional clauses is displayed in a text box below the tests.
The numbers in the expression refer to the row number in the test clauses table. So "1 AND 2" means the test clauses in row 1 and row 2 (the AND means both clauses have to pass for the overall record to pass).
You can adjust the test composition by directly editing the composite expression. Click the Edit button and type in a new expression. Experienced users will find this faster than using the Logic column dropdown options.
The Logic column and the Composite Expression field are mirror images of each other. Any changes to either are reflected in the other. For example, if I changed AND to OR in the logic column (second row), the composite expression would change to match. Conversely, if I changed the composite expression to 1 AND ( 2 OR 3 ), then the test clauses would change to match, including deleting the now unused 4th row.
If a Tester or TestFilter has a data cache, you can access a list of cached values to fill in your tests. This "data-aware" transformer dialog is available in the Tester, TestFilter, AttributeManager, and AttributeExposer.
Operators
The list of operators available in the Tester transformer (or in many of the other locations that make use of the Tester dialog) looks like this:
Besides the usual operators, some are based on a SQL WHERE clause. These include:
...plus there are other tests that check for the existence of attributes and values:
"Attribute has a value" is the opposite of the three other tests; for example, this attribute is not Null, AND it is not an empty string, AND it is not missing. Incidentally, "missing" means the attribute does not exist on the tested record.
"Contains Regex" means only part of the string needs to match. For example...
Attribute Value: abcd
Search String: ^ab
Contains Regex: PassedFor example, the entire string doesn't need to match.
The TestFilter allows multiple conditions, each with multiple test clauses. Each condition is given an output port, with an additional output port for records that fail all test conditions.
The TestFilter is similar to the CASE or SWITCH command in programming or scripting languages. In Workbench, it looks like this:
Notice that there are multiple conditions and an output port for each. Each condition/port combination is equivalent to a single Tester transformer; hence, the TestFilter is an excellent way to combine multiple Tester transformers into one.
The TestFilter output ports can be given custom names rather than Passed/Failed, which is another advantage to this transformer over the Tester.
The TestFilter has the complete set of operators available with the Tester, such as equals, greater than, less than, and so forth. Each condition is tested in turn.
Records that pass are output through the matching output port. Records that fail are sent to the following condition in the list. Therefore, it’s essential to get the conditions in the correct order.
The TestFilter is very good for filtering a record by a set of cascading conditions; for example, here are tests to determine whether to send out a snow plow:
- Has there been more than 150 mm of snowfall?
- Has there been more than 100 mm of snowfall, AND is this an Arterial road?
- Is the temperature less than zero degrees Celsius, AND was sand last applied more than 24 hours ago?
It’s a set of cascading tests because if there has been more than 150 mm of snow, the plows are sent out anyway; you don’t need to test any other criteria. So, the test order is significant. If every test fails, then the plows are not sent out. If using the above snowfall example, you could use a single TestFilter instead of three Tester transformers.
Because the TestFilter can carry out a single test (as well as multiple ones), it's possible to use it exclusively instead of the Tester transformer.
While using the TestFilter, you can choose to output all records through the First Passing Port or All Passing Ports using the Test Output dropdown.
⭐ New for FME 2025.1: the TestFilter now has a new parameter called “Output Attribute Name” in the Advanced parameters section. You can choose a name for this attribute and it will store the name of the output port as a value. This option lets you filter your results and store the result in an attribute for accessing later, allowing you to avoid the repetitive pattern of using many AttributeCreators after a TestFilter.
Sven, a Planning Analyst, has been asked to gather park maintenance data to help understand the impact of a proposed change to park funding in the Downtown neighborhood. He has a MITAB dataset of parks and needs to filter the data to estimate the proposed changes' cost accurately.
We need to remove dog parks from the data because they have their own costs. We can do this with a Tester transformer.
After using the Tester, we realize we'd be better off using a TestFilter here. We would like to separate dog parks, non-dog parks that have a name, and unnamed parks. The TestFilter will let us do that in one transformer instead of using multiple Testers.
We can process the Dog Parks and Unnamed Parks streams later.
